Operator independent image cytometer

ABSTRACT

An operator independent image cytometer having a method for image segmentation. Image segmentation comprises the steps of filtering a digital image of a cellular specimen and thresholding the resultant image. In addition, the thresholding may include the sorting of features extracted from the filtered image. The present invention also includes a method for cytometer autofocus that combines the benefits of sharpening and contrast metrics. The present invention further includes an arc lamp stabilization and intensity control system. The image cytometer has broad applications in determining DNA content and other cellular measurements on as many as 10 5  individual cells, including specimens of living cells. Image segmentation applications include PAP smear analysis and particle recognition.

STATEMENT REGARDING GOVERNMENTAL RIGHTS

This invention was made with support from the United States Governmentunder PHS Award/Grant No. 2 T32 HL 07089-16 awarded by the Department ofHealth and Human Services. The Government has certain rights in theinvention.

This application is a continuation of application Ser. No. 08/017,321,filed Feb. 11, 1993, which is hereby abandoned, which was a continuationof application Ser. No. 07/729,383 filed Jul. 12, 1991, which isabandoned.

MICROFICHE APPENDIX

A microfiche appendix containing computer source code is attached. Themicrofiche appendix comprises one (1) sheet of microfiche having 43frames, including one title frame.

The microfiche appendix contains material which is subject to copyrightprotection. The copyright owner has no objection to the reproduction ofsuch material, as it appears in the files of the Patent and TrademarkOffice, but otherwise reserves all copyright rights whatsoever.

BACKGROUND OF THE INVENTION

1. Field of the Invention

The present invention relates to image processing systems and, moreparticularly, to an image cytometer which performs image separation on aspecimen.

2. Description of the Prior Art

An automated system for analysis of anchorage-dependent cells maycontribute to improved understanding of many cellular functions. Invitro, anchorage-dependence is defined by the requirement that cells beattached to a substrate to proliferate. Many cell types exhibitanchorage-dependence and the loss of this requirement for attachment isusually associated with malignant transformation. (See, e.g., R. I.Freshney, "The Transformed Phenotype," in Culture of Animal Cells, aManual of Basic Technique, 2nd ed., New York: Alan R. Liss, pp. 197-206,1987.) In vitro, cell division, cell shape, cell migration, and controlof cell growth and differentiation are all at least altered byinteraction of cells with a substrate. For a fully automated cytometerto have the potential to analyze cellular parameters that depend oncontact of cells with a substrate, the system would have to performanalyses in situ. The basis for such a system is analysis of images ofthe cells acquired from a computer-controlled microscope. The imagesacquired by scanning a specimen of cells would be analyzed to performcytometry, the measurement of individual cells. The instrument thatperforms this task, the scanning or image cytometer, would ideallyindicate: 1) the size and shape of cells, nuclei and key organelles; 2)the distributions and concentrations of important cellular substances;and 3) the organizational relationships of cells. Such an instrumentshould scan a specimen rapidly, nondestructively, and repeatedly toanalyze the dynamics of large numbers of cells.

There are many potential applications for scanning or image cytometry.Cell division, for example, could be analyzed directly by repeatedlyscanning a large population of cells as they progress through the phasesof the cell division cycle. Current methods for measuring durations ofthe cell cycle phases include bromodeoxyuridine pulse labeling in flowcytometry and time-lapse cinematography in microscopy. Flow cytometrypermits rapid analysis of large numbers of cells, but cannot repeatedlymeasure a given cell, and time-lapse cinematography allows repeatedmeasurements of the same cells, but cannot be used for more than one ora few cells.

The scanning cytometer should bridge this gap, making it possible totrack the lineage and cell cycle phase times of each cell in a largegroup, and differentiate between those that are quiescent and those thatcontinue to divide. It should also be possible to correlate changes innuclear size, shape and chromatin distribution with progression throughthe cell cycle. (See, e.g., C. De Le Torre and M. H. Navarrete, Exp.Cell Res. 88: 171-174, 1974; W. Sawicki, J. Rowinski and R. Swenson, J.Cell Physiol., vol. 84: 423-428, 1974; F. Giroud, Biol. Cell 44:177-188, 1982.) In addition to improving understanding of cell division,certain scanning cytometry measurements may be sensitive indices of cellhealth. If so, scanning cytometry may lead to practical, automatedassays for toxins, drugs and infective agents.

A number of techniques for cell measurement have been used with limitedsuccess. For example, one may use the "metabolic rate" method disclosedby E. A. Dawes, Quantitative Problems in Biochemistry, Baltimore:Williams and Wilkins, pp.293-311, 1972, or the "pooled quantity" methoddescribed in R. I. Freshney, "Quantitation and Experimental Design," inCulture of Animal Cells, a Manual of Basic Technique, 2nd ed., New York:Alan R. Liss, pp. 227-256, 1987. These two types of techniques are usedto analyze the culture as a whole. Measurement of the metabolic rate ofcells by CO₂ production or O₂ consumption allows analysis of live cellswithout disruption, but yields no individual cell data. There are also anumber of methods for measuring amounts of substances from the wholeculture that require cell destruction. Such chemical methods include themodified Lowry assay for proteins and DNA extraction assays.

In addition, there are automated or semi-automated devices available forcell measurement. In these devices, such as the Coulter counter and theflow cytometer (also known as the FACS or fluorescence activated cellsorter), measurements are typically made on individual cells insuspension. The Coulter counter provides the simplest individual celldata, i.e., cell number. More advanced electronic counters also measurecell size. However, the Coulter counter tends to give a count only, andhas the disadvantage of causing cellular disruption. The flow cytometergives cell number and size, as well as the quantity of cellularsubstances labeled with a specific fluorescent dye. The flow cytometerprovides very low spatial resolution, however, and also tends to disruptthe cell culture under examination. Finally, while both the Coultercounter and the flow cytometer provide data on individual cells, theycannot analyze cells attached to a substrate. There are many advantagesto measuring cells without disturbing them in their "native" location(e.g., attached to a substrate), and these advantages are not possiblewith the above-noted devices.

Measurements of individual cells in situ have been performed withvarying degrees of automation in microscopy. The simplest approaches,such as utilization of a hemocytometer, require the microscopist tocount cells and record information by hand. More complicated techniques,made possible by modifications to the microscope, allow quantitation ofspecific cellular parameters.

The first measurements of specifically labeled cellular substances wereperformed using a photomultiplier tube. Use of a photomultiplier tuberequires positioning the cell of interest under an aperture and does notallow measurements of shape and size. The combination of image analysisand microscopy yields size, shape, and pattern measurements and allowsquantitation of labeled cellular substances (see, e.g., B. H. Mayall,"Current Capabilities and Clinical Applications of Image Cytometry,"Cytometry Supplement 3: 78-84, 1988). There are numerous advantages inthe added information available from microscope images, but the methodsused to analyze them are generally more labor intensive, resulting inthe analysis of much smaller numbers of cells than with flow cytometersor electronic counters. Standard microscopic methods of cell measurementare also impaired, in that they require significant human interactionand are thus quite slow. In addition, such methods have the furtherdisadvantage of producing data that is quite subjective and is based ona limited number of cells.

Complete automation of image cytometry is necessary for practicalanalysis of large numbers of fixed cells and efficient repeated scanningof groups of live cells. A summary of some of the systems capable offully automated measurement of cellular specimens is given in Table I,below. The first of these systems counts cell colonies and measurescolony size, but returns no individual cell data. Fully automatedanalysis of cell motility has been implemented for both single cells andgroups of cells (see, e.g., G. Thurston, et al., "Cell MotilityMeasurements with an Automated Microscope System," Exp. Cell Res. 165:380-390, 1986). In that report, location was recorded with each scan butno attempt was made to analyze cell size or shape, or the quantity ofcellular substances.

Others have reported success at determining whether or not a smearcontains malignant cells with instruments capable of rapidly scanning amicroscope slide. The machine diagnosis was compared with the expertopinion of a pathologist. Presumably, the machine diagnosis was based onthe shape and density of the cell nuclei. Data such as DNA content andnuclear size compiled from the individual cells, however, was notpresented. It is, therefore, impossible to know whether the methods usedfor automated cytology might be adapted to allow precise measurements ofcell shape and size, or the quantity of cellular substances.

                  TABLE I                                                         ______________________________________                                        Previous Image Cytometry Automation                                           Application Recognition    Measurements                                       ______________________________________                                        Colony Counting                                                                           computer,      colony number,                                                 phase contrast size                                               Cell Motility                                                                             computer,      location,                                                      phase contrast movement                                           Cytology    computer,      malignant vs.                                                  fluorescence,  nonmalignant                                                   PAP, Faulgen, etc.                                                ______________________________________                                    

Measurements of the cell nucleus and DNA content have been the focus ofmany cytometric studies because nuclear abnormalities are oftenassociated with malignancy and because nuclear changes define thedifferences between the phases of the cell cycle. Other investigatorshave reported working with systems capable of various levels ofautomation for nuclear analysis. The SAMBA system has been used tomeasure DNA content on as many as 600 cells/experiment (see, e.g., E.Colomb, et al., Cytometry 10: 263-272, 1989) and the LEYTAS system hasbeen used to measure DNA content on 100-300 cells/experiment (see, e.g.,C. J. Cornelisse, et al., Cytometry 6: 471-477, 1985). The number ofcells analyzed in these experiments is much smaller than the 10⁴ -10⁵cells that can be analyzed in flow cytometry. A recent image cytometryreview (see B. H. Mayall,Cytometry Supplement 3: 78-84, 1988), whichpresented DNA content experiments with 200 cells each, identified thenecessity for operator interaction as a major impediment in the analysisof larger numbers of cells.

Image cytometry usually requires interactive selection of the objects ofinterest. During interactive operation a technician must either drawobject borders with the aid of a digitization tablet or mouse, orutilize semi-automated techniques based on intensity thresholding andediting of incorrectly chosen objects. For example, the patent to Bacus(U.S. Pat. No. 5,018,209) discloses one such operator assisted system.

An image cytometry system that can perform nuclear analysis unattendedby an operator has not yet been reported, prior to the presentdisclosure. Some review articles which provide a frame of reference forappreciating the improvements and novel features of the presentinvention are as follows: Roberts, J. NIH Research 2: 77 (1990); Herman,et al., Arch. Pathol. Lab. Meth. 111: 505 (1987); Baak, Path. Res.Pract. 182: 396 (1987); and Mayall, Cytometry Supplement 3: 78 (1988).

Accurate computer recognition of the cell nuclei in an image is thefirst step in fully automated measurement of DNA content and nuclearsize, shape and pattern. In an image of cells stained with a fluorescentdye specific for DNA, computer recognition consists of correctlysegmenting the image into bright foreground objects and dark background.There are many examples of image segmentation or cell edge findingtechniques used for computerized recognition (see, e.g., L. O'Gorman, etal., IEEE Transactions on Biomedical Engineering 32: 696-706, 1985). Itis difficult, however, to compare the performance of these differenttechniques because each method was developed for a specific applicationand demonstrated on only one or a few images. An assessment of thereliability of these techniques on large numbers of cells, withpresentation of measurements such as DNA content, was not provided. Thesimplest of these methods, intensity thresholding, has been evaluatedfor measurement of the DNA content of fluorescent stained smears (see,e.g., T. Takamatsu, et al., Acta Histochem. Cytochem. 19: 61-71, 1986.).Thresholding resulted in lower precision than attained by flowcytometry. In that report, unreliable recognition was identified as aprobable source of error.

In the field of image processing, image segmentation, i.e., theautomated separation of objects from a background in a digital image, isa recurring theme. Previous methods for image segmentation, or objectrecognition, have included thresholding (or clustering), edge detectionand region extraction (K. S. Fu and J. K. Mui, "A Survey on ImageSegmentation", Pattern Recognition, vol 13, pp 3-16, 1981).

In thresholding, the computer utilizes differences in image intensity todelineate features from a background. In its simplest form, the image isthresholded into two intensity ranges. All pixels (or picture elements)below the threshold intensity value are separated into one group whileall those equal to or above that value are separated into a secondgroup. In more complicated methods, multiple thresholds are used and thethreshold values are determined by a method called "clustering." Eachset of intensity ranges can be used to identify a different type ofobject if intensity differences are well defined. Difficulties arisewhen the objects contain a broad range of intensities, or when objectedges are characterized by gradual, rather than abrupt changes frominternal intensity to background intensity.

In edge detection, the edges of the objects are assumed to occur wherethere are large changes in intensity within a short distance (smallneighborhood of pixels) in the image. These steep intensity gradientscan be enhanced by edge filters (convolution or Fourier). After thefilter is applied, the edges appear either as white pixels on a blackbackground or black pixels on a white background. Thresholding can thenbe used by a processor to locate the edge pixels. These edge pixels mustthen be connected by the processor and sorted to form separate boundaryrepresentations of each individual object in an image. The mostcomplicated (and processor intensive) step involves connecting the edgepixels into continuous boundaries. In many objects, filtering results indisconnected and spotty edges that are difficult to connect andsometimes result in the joining of separate but close proximity objects.One such edge detection system was disclosed by Martin (U.S. Pat. No.4,561,104).

Region extraction methods depend on searching sets of image pixels forsimilarities and grouping them according to predetermined criteria. Thismethod sequentially searches arbitrary regions of the image forsimilarities or differences. If two adjacent regions are similar theyare merged and if a single region is found to contain too muchvariability it is divided. Region merging and/or division is carried outrepeatedly until the algorithm determines that the image has beensegmented as well as possible. The difficulty with this method is infinding similarity criteria for grouping that can be easily implementedby computer. The other problem is that the repeating (iterative) natureof these methods can make them too slow for all but the fastest, mostexpensive computers.

In addition, image segmentation would be enhanced if an image cytometerhad a stable source of light. Arc lamps are known to emit intensity thatvaries. Some of this time varying intensity is due to arc wander. Arcwander means that the source of the luminescence changes position withtime. Arc wander causes the light cast on the microscope specimen tochange intensity in a way that depends on the location within themicroscope field of view. This means that one spot in the microscopefield can increase in intensity while another spot simultaneouslydecreases in intensity. Due to this spatially dependent intensityvariation it is not possible to place a light measuring device, such asa photodiode, at any one place in the light beam and utilize its signalto correct for intensity variation. It has been previously observed (G.W. Ellis, "Microscope Illuminator with Fiber Optic Source Integrator,"J. Cell Biol. 101:83a, 1985) that placing an optical fiber in themicroscope light path scrambles the light so that spatially dependentvariation no longer exists.

Another problem with fluorescence microscopes, however, is theaccumulation of enough light to cause a visible fluorescence in thespecimen to be studied. Specially designed, aberration corrected opticalelements are usually used to transfer illumination from an arc lamp tothe microscope field. Because light transmission in optical fibers isnot perfect, use of a fiber-optic light scrambler, such as an opticalfiber in the light path of the microscope, further decreases the amountof light available for fluorescence excitation.

Light measurement in fluorescence image cytometry is also prone to errorbecause current camera and image processor systems acquire and operateon 8-bit images. An 8-bit intensity measurement incorporates 256discrete intensity values, 0 through 255. This is not enough toencompass the entire range of intensity found in an image offluorescent-stained cell nuclei. In practice, some intensities in thebrightest nuclei are actually greater than 255 and these values areincorrectly recorded as 255. This results in an underestimation of someof the integrated intensity measurements by the instrument. One way tocorrect this problem is to decrease the light intensity by a knownamount and remeasure the portion of the microscope field that was toobright. This can now be done only by mechanically introducing a filterinto the light path. The problem is that arc lamp intensity changes withtemperature and a change in the electric current powering the arc lampis followed by a slower change in temperature. This makes it essentiallyimpossible to control arc lamp intensity by altering current alone.

Therefore, what is needed is a degree of automated image segmentationfar greater than that achieved via use of all presently known imagecytometers, including those described above. In particular, it would bedesirable to analyze a significantly greater number of cells than ispossible with other devices, requiring only minimal operator interactionat the very beginning of the procedure and data organization at the end,with complete operator independence during the actual measurementprocess. Furthermore, it would be an advantage if an image cytometercould stabilize and control the intensity of an arc lamp.

SUMMARY OF THE INVENTION

The above-mentioned needs are satisfied by the present invention whichincludes a method for locating objects in an image. This method utilizesa digital filter, such as convolution or Fourier, to create a secondimage in the computer from which objects can be recognized bythresholding an image parameter such as intensity.

The method, termed pattern filtering object detection, assures that theimage objects to be detected by the computer contain complicatedpatterns that are different from the background. Using the appropriatefilter, it is possible to transform the object patterns into a high (orlow) intensity and the background patterns into a low (or high)intensity. Following the conversion of the image into an intermediate,high object-to-background contrast image, thresholding is used toseparate the object pixels from background pixels and the object pixelscan be sorted into individual object structures in computer memory. Thismethod has an advantage over simple edge detection because moreinformation within the object is utilized to determine its extent thanjust the edge region. It also has a distinct advantage over regionextraction because region similarities are enhanced in a single step,rather than a repeating, iterative fashion. Fast computer hardware forimplementing convolution and Fourier filtering is just becominginexpensive enough to make this method practical.

In Fourier spectrum terms, each object may be thought of as having aspecific set of frequencies. This set of frequencies can be transformedinto an image where this set of frequencies is represented by a high (orlow) intensity while the background, which contains a different set offrequencies, is transformed into a low (or high) intensity. The betterthe filter performs, the wider will be the intensity difference betweenthe objects and background and the greater the intensity gradients atthe boundaries.

If filtering does not create high enough object-to-background contrast,other methods can be used to improve image segmentation. In theimplementation described here, the filter makes it possible to selecteach object with a single threshold but objects are sometimes differentenough from each other to require different thresholds for objects inthe same image. Therefore, a method for adjusting the threshold for eachindividual object is utilized to improve the object detection.

An additional problem arising from a less than ideal filter is theenhancement of internal edges to the degree that some pixels inside theobject fall on the incorrect side of the final threshold. This problemsometimes occurs with the cell nucleus images segmented in the imagecytometer application described herein. Since the pixels erroneouslyidentified as background are always completely surrounded by objectpixels, they can be corrected by a hole filling method that converts allcompletely enclosed background pixels into object pixels. The holefilling method is non-iterative and fast.

One goal of the present invention is the development of a system capableof unattended analysis of cell monolayers stained with a dye or otherlabel. In one embodiment, the label is specific for DNA. In variousembodiments, the label is a stain or dye; in others, it is a fluorescentdye. In preferred preparations of specimens, overlap artifacts are rareand there are few sources of nonspecific fluorescence. In addition, thecells are relatively flat, decreasing error due to limited depth offield. Along with the high contrast provided by fluorescent staining,these characteristics simplify image analysis and identification ofphotometric errors due directly to the instrument rather than thespecimen. The discussions and data presented herein for the presentimage cytometer include methods for fully automated operation andprecision measurement of fluorescent light.

The present invention also includes a novel arc lamp stabilization andintensity control system. It has been found that the removal ofspatially dependent intensity changes by a fiber optic light scrambleralso allows a photodiode-feedback circuit to be used to stabilize theremaining temporal fluctuations. This becomes possible because after thefiber optic scrambler, intensity changes in any portion of the lightcoincide with changes in all portions of the light.

A device for increasing the amount of light transmitted into an opticalfiber is currently available. This device (Photomax arc lamp, from OrielCorp. of Stratford, Conn.) uses an ellipsoidal mirror to collect 2-4times as much light from an arc lamp as the lens and mirror system usedin commercially available microscope arc lamp sources. Becausemicroscopes are thought of as image forming devices, the idea ofutilizing an ellipsoidal reflector in the illumination system isunconventional. However, because the optical fiber is utilized as alight scrambler, its output pattern is substantially independent of theinput pattern. Therefore, the fact that the ellipsoidal reflector mayhave aberrations and is not a good image forming optical element isunimportant. Only the amount of light transferred to the optical fiberis important. In addition to the increased illumination with anellipsoidal reflector, the optical configuration is simplified. Whileconventional microscope arc lamps require a reflecting mirror and twolens groups to focus the light onto the optical fiber, the light fromthe ellipsoidal reflector, as used in the present invention, focusesdirectly onto the optical fiber with no intervening lens elements.

In conjunction with the fiber optic light scrambler and thephotodiode-feedback stabilization described above, disclosed herein is adevice for electrical control of arc lamp intensity. This involvesutilizing a servo controlled amplifier that matches the intensity of thelamp to a computer-controlled reference value. The servo amplifierstabilizes the light intensity by matching the voltage from thephotodiode to a predetermined reference voltage. This reference voltageis controlled by the computer and can be changed to alter lightintensity. It has been found that the intensity can be controlledthrough at least a ten-fold range without extinguishing the arc lamp.

These and other objects and features of the present invention willbecome more fully apparent from the following description and appendedclaims taken in conjunction with the accompanying drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a block diagram of one presently preferred embodiment of thecytometer of the present invention;

FIG. 2 is a representation of a magnified image of cells as seen throughthe microscope of the cytometer shown in FIG. 1;

FIG. 3 is a three-dimensional plot of a gray-scale object that isrepresentative of a cell;

FIG. 4 is a block diagram of the presently preferred image processor ofFIG. 1;

FIG. 5 is a flow diagram of the computer program that controls the imagecytometer of FIG. 1;

FIG. 6 is a flow diagram of the image separation portion of the flowdiagram shown in FIG. 5;

FIG. 7 is a three-dimensional plot of the object shown in FIG. 3, afterconvolution filtering;

FIG. 8 is a three-dimensional plot of the object shown in FIG. 3, afterimage separation;

FIG. 9 is a flow diagram of the Sort II portion of the flow diagramshown in FIG. 6;

FIG. 10 is a DNA content histogram and average area plot illustratingcorrected and uncorrected data from image cytometry;

FIG. 11 is a three-dimensional plot illustrating the number of nuclei,the nuclear area, and the DNA content of a cell culture;

FIG. 12 illustrates results of a DAPI toxicity assay on normal humanneonatal foreskin fibroblasts;

FIG. 13 is a schematic diagram of the optics of an epifluorescentmicroscope of the prior art;

FIG. 14 is a schematic diagram of the optics of the epifluorescentmicroscope shown in FIG. 13 as modified by a fiber optic scramblerassembly of the prior art;

FIG. 15 is a schematic diagram of the arc lamp stabilization andintensity control system of the present invention; and

FIG. 16 is a schematic diagram of the photodiode amplifier of FIG. 15.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

Reference is now made to the drawings wherein like numerals refer tolike parts throughout. The following detailed description is dividedinto four sections: The Image Cytometer, Applications of the ImageCytometer, The Arc Lamp Stabilization and Intensity Control System, andSummary.

The Image Cytometer

FIG. 1 illustrates the presently preferred embodiment of an operatorindependent image cytometer 100 of the present invention. The hardwarecomponents of the cytometer 100 include an epifluorescent microscope102, a motorized stage 103, controlled by a pair of XY motors 104a and aZ motor 104b, XYZ stage controller 106, a video camera 108 (preferablymodel NC200, Patterson Electronics of Irvine, Calif.), an imageprocessor 110, and a host processor 112.

The microscope 102 (individual parts not shown) is preferably a NikonOptiphot with a 100 watt mercury vapor arc lamp, a quartz collector anda 20x fluorite objective for UV fluorescent excitation. The fluorescentfilter block (Omega Optical of Brattleboro, Vt.) consists of anexcitation filter with a peak transmittance at 365 nanometers and a 10nm bandwidth, a dichroic mirror that reflects below 405 nm, and abarrier filter with a peak transmittance at 485 nm and a 30 nmbandwidth.

The host processor 112 is preferably a microcomputer such as an IBMPC/AT compatible having an Intel 80286, 10 MHz, 1 wait state CPU, and a2 Megabyte RAM memory and 80 Megabyte hard drive (not shown), availableas a unit from Datel (San Diego, Calif.). The host processor 112controls the image processor 110 (preferably a Series 151 from ImagingTechnology Inc. of Woburn, Mass.) and the motorized stage 103 (such asone available from Syn-Optics of Sunnyvale, Calif.). The host processor112 communicates with the image processor 110 via an interface board(supplied with the Series 151) that is plugged into an expansion slot inthe host processor 112. The host processor 112 communicates with thestage controller 106 via a serial port to move the stage 103 in the X, Ydirections for lateral positioning and the Z direction for autofocus.The stage 103 is moved under the control of the host processor 112 sothat various portions or fields of a specimen 114 can be examined.

A portion of an example specimen, such as the specimen 114 of FIG. 1, isshown in FIG. 2. FIG. 2 represents a magnified image of a specimencomprising a set of cells, particularly cell nuclei, generally indicatedat 116. In this example, the cells 116 are normal neonatal foreskinfibroblasts that have been plated on a washed, autoclaved microscopeslide (not shown). The cells 116 have been stained with a fluorescentstaining solution as will be further discussed hereinbelow.

The fluorescent staining produces increased light intensity in the cellnuclei. The representation of FIG. 2 shows the cells, or cell nuclei116, in a reverse or negative image as darker regions against a lightbackground 118. However, it should be understood that the positive, or"normal", image will have the cells 116 appear as light regions againsta dark background. Henceforth, a reference to an image will refer tosuch a normal image.

It should be noted that the cells 116 do not share the same intensityfrom one cell to another, or even from one point within any single cellto another. Hence, segmenting the cells 116 from the background 118 forfurther processing by a computer cannot be performed by using only anintensity thresholding technique.

FIG. 3 shows a three-dimensional plot of a gray-scale digital image of acell (such as one of the cells 116 shown in FIG. 2, but here the cell isshown in its normal image form of higher light intensity on a lowerintensity background). Note that after the image of one or more cells isreceived by the video camera 108 and digitized by the image processor110, each digitized cell is then referred to as an object 120. The areasurrounding the object 120 is termed a background 122.

The X, Y plane of the plot corresponds to the X, Y plane of the stage103 (FIG. 1). The Z, or vertical, axis represents light intensity. Theplot is divided into small units commonly referred to as pixels as isindicated in FIG. 3, for example, by a pixel 124. A scaling spike 126,representing maximum intensity, is located at one corner of the plot.The plot clearly shows the variation of the intensity commonly foundwithin a single cell.

A fundamental problem that is addressed by the present invention isimage separation, i.e., separating many objects, such as 120, from theimage background so that the cells 116 (FIG. 2) can thereafter beanalyzed by a computer. The process of image segmentation begins whenthe magnified image is fed from the CCD camera 108 (FIG. 1) to the imageprocessor 110.

A block diagram of the preferred image processor 110 is illustrated inFIG. 4. It should be observed that while an image processor willgenerally speed up the image segmentation of the present invention,there is no reason why the calculations performed therein could not takeplace in the host processor 112 (FIG. 1) or any other computer.

The image processor 110 is preferably configured with five functionalunits, or boards, as follows: (1) an analog/digital interface (ADI) 130,for analog-to-digital conversion of the RS-170 video signal generated bythe camera 108 (FIG. 1); (2) a frame buffer (FB) 132, for storage of one16-bit and two 8-bit 512×512 digital images; (3) the histogram/featureextractor (HF) 134, for creating histogram and feature arrays; (4) anarithmetic & logic unit (ALU) 136, for multiplication, addition,subtraction, logical operations, and bit shifts; and (5) a real timeconvolver (RTC) 138, for 4×4 or smaller convolutions.

The preferred image processor 110 performs all the above operations inreal time (1/30th seconds), or faster in area-of-interest (AOI) mode.AOI mode allows the selective processing of only a portion of a digitalimage. The time required for AOI mode operations is proportional to thenumber of pixels, or picture elements, in the selected region.

Understanding the basic mechanisms by which the five image processorboards 130, 132, 134, 136, 138 communicate and function is important forunderstanding the present invention. Image operations, such assubtraction, multiplication, and convolution are carried out by the ALU136 and RTC 138. The ALU 136 and RTC 138 are pipeline processors. Thismeans that image information flows into these boards 136, 138, isoperated on, and flows out. The image information is always flowing. Ifthe ALU 136 is set up for multiplication of two images stored in the FB132, then one multiplication is occurring every 33 milliseconds as longas the set-up remains and the image processor 110 is powered on. Controlis maintained by having the host processor 112 instruct the FB 132 toacquire the information coming from the processors 130, 136, 138. Fromthe point of view of the FB 132, information flows out over three buses,video data A (VDA) 140, video data B (VDB) 142, and overlay (OVR) 144,and in over two buses, video data in (VDI) 146 and video pipeline in(VPI) 148. The FB 132 is always broadcasting information over its outputbuses and information is always available to it over its input buses. Ifthe instruction to acquire is not sent to the FB 132, the results of theoperations are not stored. Programming the operations of the boards inthe Series 151, therefore, is a matter of controlling the flow of imageinformation as well as setting specific operations on or off.

The frame buffer 132 contains 1 Megabyte of random access memoryorganized as two 8-bit×512×512 image stores called, respectively, B1150a and B2 150b, and one 16-bit×512×512 image store called A, orFRAMEA, 152. FRAMEA 152 can also be treated as two 8-bit images. The VDA140 continuously carries the 16-bit information stored in FRAMEA 152 andthe VDB 142 continuously carries 8-bit information stored in either B1150a or B2 150b. A multiplexer (not shown) controls which image iscarried by the VDB 142, i.e., the image stored in B1 150a or B2 150b.Control over which image is operated on is maintained at the input tothe pipeline processors 136, 138. The image output from the pipelineprocessors 136, 138 is available only on the 16-bit VPI 148. Thisprocessed VPI image information can be acquired directly only by FRAMEA152. The 8-bit overlay bus (OVR) 144 is used to create an overlay (fordisplay of nuclei edges on a monitor that is not shown) on the imagesstored in FRAMEA 152 and B1 150a using information stored in B2 150b.

The analog/digital interface 130, the primary responsibility of which isconversion of the analog video signal (from the CCD camera 108 inFIG. 1) to digital format, also acts as a simple pipeline processor. Ithas access to the VDA and VDB buses 140, 142 and can perform look-uptable transformations on information from these buses and broadcast thetransformed images over the VDI 146. The 8-bit VDI image information canbe acquired directly by B1 150a or B2 150b, and indirectly by FRAMEA 152through the pipeline processors 136, 138. The VDI 146 also carries theimages acquired from the camera inputs, one of which is used for the CCDcamera 108. Image transfer from B1 or B2 150 to FRAMEA 152 must beperformed through the ALU 136 (with or without processing) andinformation from FRAMEA 152 can be transferred to B1 or B2 150 throughthe ADI 130.

Information can also be transferred from the Series 152 image processor110 to the host processor 112. In addition to reading image informationin the form of pixel intensities, most of the registers (not shown) ofthe image processor 110 can be read to determine the operationscurrently set. Processed image information is available from only twosources: the ALU 136 min/max registers and the HF 134. The ALU 136 candetermine the minimum and maximum intensities in an image and the HF 134provides more complicated processing, histogram compilation and featureextraction. The HF 134 provides no pipeline processing. Images read bythe HF 134 are converted into information read only by the hostprocessor 112. There are no image output buses carrying images alteredby the HF 134.

Real-time histogram and feature extraction capabilities of the imageprocessor 110 (FIG. 1) are important for timely operation of thecytometer 100. The histogram array (not shown), generated by the HF 134in histogram mode, is an array containing the number of pixels in theimage at each intensity (e.g., for an 8-bit pixel, gray-scale image, theintensity range is 0, representing minimum intensity, to 255,representing maximum intensity). The histogram can be used for intensitystatistics. For example, obtaining the average and standard deviation inthe image for the purpose of autofocus. In feature extraction mode, theHF 134 provides an organized array of all pixels at defined sets ofintensities. As will be further discussed below, the groups of pixels or"streaks" are compressed by the HF 134 using the well-known method ofrun-length encoding (RLE). The Series 151 is programmed by writing toregisters on the processing boards. A set of higher level routines isprovided by the Series 151 Library.

FIG. 5 illustrates the process for controlling the operator independentcytometer 100 of FIG. 1, beginning at a start state 160. Prior tostarting the scanning cytometry program, the scanning area is defined,the shade correction image is calculated (scanning area and shadecorrection are discussed below), and the gain and offset on the imageprocessor and camera are set. Gain and offset are adjusted with the aidof an oscilloscope to read the analog video signal and a histogramoverlay to view the range of image intensities. (The histogram overlayis a graphical plot of pixels number vs. intensity, created from thehistogram array provided by the image processor 112. This plot isoverlayed on the image displayed on a monitor.) These adjustments aremade to set the background to zero intensity and ensure that theintensities fall as much as possible within the measurement range of thesystem (presently an 8-bit range of 0 to 255).

Software for the preferred image cytometer 100 was written using theImaging Technology 151 Library, Version 2.3, and the Microsoft CCompiler, Version 5.1. The 151 Library contains routines for basichardware control of the image processor 110. In order to develop thesoftware detailed in the attached Microfiche Appendix, a libraryconsisting of higher level subroutines was written. Both libraries wereused to write a Series 151 interactive command program for subroutinetesting prior to incorporation into the image cytometer program, whichresides on the host processor 112. Nonetheless, one skilled in thetechnology will recognize that the steps in the accompanying flowdiagrams can be implemented by using a number of different compilersand/or programming languages.

From the start state 160, the cytometer 100 moves to a state 162 toset-up the first field. The scanning area for a 20x objective may, forexample, comprise 8,000 fields, or 512×480 pixel images, that are eachapproximately 250×330 microns. The motorized stage 103 (FIG. 1) is movedto a first field and the microscope 102 is then focused manually for aninitial, rough focus.

Moving to a state 164, the cytometer 100 tests whether the field underconsideration contains any cells. Movement to a new field occurs at astate 166 if image intensity is too low to contain a nucleus (or whenanalysis of one field is complete). For example, if there are less than801 pixels of intensity greater than 35, autofocus is not performed.This number of pixels is calculated from the image histogram. Bydefinition, adjacent fields do not overlap and nuclei touching the imageborder are ignored. If an image is bright enough to contain a nucleus,then the cytometer 100 proceeds from the decision state 164 to anautofocus state 168.

Autofocus is a requirement for any fully automated microscope-basedimage processing system. Autofocus is necessary because of the smalldepth of field in the microscope 102 (FIG. 1), typically on the order ofa micron. It is more practical to perform autofocus on each new fieldthan to achieve the otherwise required flatness over a few squarecentimeters. For this reason, various autofocus functions were tested todetermine those best suited for image cytometry.

Note that autofocus is controlled from the host processor 112 (FIG. 1).The host processor 112 can perform a transformation on the image toobtain a value which represents a degree of focus. This value can thenbe compared with another value obtained from another image after thestage 103 is moved up or down via the XYZ stage controller 106.

There are several fundamentally different methods in use for autofocus.Most of these methods fall into two categories: position sensing andimage content analysis. Position sensing methods, such asinterferometry, require independent evaluation of the best focuslocation and, more importantly, a single well-defined surface from whichto reflect light or sound. In biologic specimens there are usually tworeflective surfaces, the coverslip and slide. In addition, tissuespecimens with significant depth lie on a slide and best focus is notnecessarily achieved at the surface of the glass. These problems makeabsolute position sensing methods impractical for use in lightmicroscopy. Image content analysis functions, on the other hand, dependonly on characteristics measured directly from the image.

The cytometer 100 (FIG. 1) of the present invention uses an imagecontent analysis function for autofocusing the microscope 102. Bestfocus is found by comparison of an image characteristic between a seriesof images acquired at different vertical positions. This method ofautofocus requires no independent reference and, as long as thecoverslip is clean, is not affected by the second reflective surface.Its most important limitation is speed, which is dependent on the videorate of the camera 108, the time for repositioning the stage 103, andfunction calculation time in the image processor 110 and the hostprocessor 112.

Conceptually, autofocus functions are based on the observation thatimages increase in contrast and image sharpness as focus improves. If animage consists of light and dark regions, the light regions becomedarker and the dark regions become lighter as the microscope 102 ismoved farther from focus. This change in contrast can be describedmathematically by the change in variance or standard deviation of pixelintensity, i.e., an increase in contrast corresponds to an increase instandard deviation.

If an image contains discrete objects with well defined edges, the edgesblur as the image moves out of focus. Image sharpness can be measured byanalyzing the Fourier frequency spectrum of the image, or by theapplication of gradient filters that isolate higher frequencies in theimage. The magnitude of the high frequencies or gradients can then beused as a measure of best focus. Either of these magnitudes, or acombination of thereof, is obtained using a selected autofocus function.

The specimens utilized thus far for the image cytometer 100 consist of afixed monolayer of cells stained with a fluorescent dye. The next stepin complexity is the analysis of live fluorescent-stained cells. Becausefluorescent excitation, especially with UV light, is toxic to cells, itis best to limit the exposure as much as possible. With acomputer-controlled shutter the exposure can be limited to <66 ms forthe acquisition of each image. Autofocus has been found to require thetesting of a minimum of about seven positions for reliable focus. If thefluorescent image is used for focusing, eight times more exposure tolight is required than for acquisition of a single image. One possiblemethod for reducing the exposure is to perform autofocus using phasecontrast microscopy, and subsequently acquire a single fluorescent imagefor analysis. Therefore, in addition to evaluation of autofocus usingfluorescent images, autofocus functions for phase contrast images havealso been studied by the present inventors. More details of differentapproaches for autofocus can be found in the doctoral dissertation ofJeffrey H. Price entitled Scanning Cytometry for Cell Monolayers,University of Calif., San Diego, 1990, which is hereby incorporated byreference.

One presently preferred autofocus function, termed F15 in theincorporated dissertation, has been found satisfactory for theapplications of the image cytometer 100 attempted to date. The autofocusfunction F15 is a combination of the 3×3 Laplacian (a well-understoodfunction for measuring image sharpness) and the variance (awell-understood function for measuring image contrast). An intermediateimage was formulated with the convolution kernel as follows: ##EQU1##and the variance was calculated from this image. In the preferredcytometer 100, the convolution kernel is fed to the real-time convolver138 (FIG. 4) and the variance of the filtered image is calculated by thehost processor 112 via the histogram array generated by the HF 134. Thesharpening filter (1) is equal to the sum of the 3×3 Laplacian and theoriginal image. The idea behind this function is to combine the effectsof gradient functions, which measure edge sharpness using first orsecond derivatives, with the effects of contrast measurement functions,which utilize the variance or standard deviation of image intensity.

F15 has been used for autofocus over 65,000 times on images of fixed,4',6-diamidino-2-phenylindole dihydrochloride (DAPI) stained cells. Thisfunction has been observed to be fast and reliable. It was implementedwith the convolution results set to 8-bit positive. The algorithm usesthe well-known binary search algorithm to move the stage 103 (FIG. 1)and locate best focus. The search range is fixed and the center of therange is the best focus at the previous microscope field. In thepreferred cytometer 100, the combination of stage movement to anadjacent field and autofocus requires about 1 second.

After autofocus, the image cytometer 100 proceeds to a state 170 to"snap", or acquire, a new image, i.e., obtain a digital image from theCCD camera 108 via the analog/digital interface 130 (FIG. 4), and shadecorrect the image. Each time an image is acquired for analysis, it mustbe shade corrected to compensate for uneven illumination. Shadecorrection is performed by multiplying the new image with a correctionimage which is prestored in the host processor 112. The shade correctionimage is calculated from a flat-field image.

A flat field is preferably created by adding 5 μg/ml DAPI and 1 mg/mlDNA (oligonucleotides from herring sperm, D-3159, Sigma, St. Louis,Miss.) to a buffer solution containing 10 mM TRIS, 10 mM EDTA, 100 mMNaCl, and 1% 2-mercaptoethanol. This homogeneous solution fluorescesevenly, providing an image that would be uniform if the light source andoptics were perfect. This solution is placed in a depression machinedinto an acrylic block and viewed through a coverslip sealed with vacuumgrease. To minimize variation due to imperfections in the coverslip, theobjective-coverslip distance is made as small as possible and the stageis moved laterally during a 256 frame average.

Shading distortion is corrected by multiplying each pixel in the newimage by the ratio of the average flat-field intensity to the flat-fieldpixel value, as is known in the technology. This operation was performedby the ALU 136 (FIG. 4) using an image calculated from the flat-fieldimage. The ALU bit shifter (not shown) allowed this floating pointcalculation to be carried out in conjunction with the ALU integermultiplier (not shown). The shade corrected image was calculated by thehost processor 112 (FIG. 1) in floating point arithmetic, bit-shiftedleft by seven bits, rounded and stored. The shade correction was appliedby multiplication of this image with each new image. A 7-bit rightrotate recovered the truncated result after multiplication and preservedthe remainder bits. Preservation of these bits allowed rounding by useof the ALU compare operation and conditional look-up tables. Althoughabout a minute is required for calculation of the shade correction imageprior to starting the image cytometer program (FIG. 5), the subsequentshading corrections are each performed in real-time.

In FIG. 5, after shade correction of the digital image, the imagecytometer 100 moves to a recognition, or image separation, function 172.Recognition is the conversion of the array of pixels making up a digitalimage into an accurate, easily accessible representation of the imageobjects in computer memory.

The simplest way for a computer to identify pixels is by differences inintensity, i.e., in a continuous tone or gray-scale image. DAPI stainedcells (further discussed below) create images of high contrast,facilitating recognition. Even with this high contrast, however, it isnot possible to accurately recognize all nuclei by a single intensityrange. This is due to the fact that the edges in images often exhibit agradual, rather than abrupt change in intensity from object tobackground. The immediate background of brighter nuclei is often equalto or greater than the intensity of dimmer nuclei. If the threshold islow enough to include the dimmest nuclei, the selection of the brightestones contains a significant number of background pixels, or imagepoints.

This problem is overcome by two methods: digital filtering and objectintensity dependent thresholding. The three main steps of therecognition function 172 of the present invention are: 1) application ofa digital filter, such as convolution, Fourier, etc., to the image thuscreating an intermediate feature extraction image, 2) first selection ofeach nucleus with a single threshold, and 3) reselection, or secondselection, of each nucleus using a local threshold calculated fromaverage intensity of the nucleus found in the first selection. Thesesteps will be further discussed below with reference to FIG. 6.

After the recognition, or image segmentation, of a field, the imagecytometer 100 continues to a state 174 to store the object data on thehard disk (not shown) of the host processor 112. If, at the subsequentdecision state 176, it is determined that more fields of the specimen114 (FIG. 1) need to be processed, then the image cytometer programproceeds to state 166 to begin another cycle with a new field.Otherwise, if all fields have been processed, the program terminates atan end state 178.

The image segmentation method of the present invention is illustrated bythe flow diagram of FIG. 6. The image segmentation function 172 is apart of the image cytometer process as shown in FIG. 5. The filter, thefirst selection, and the second selection, form a method for findingobject pixels in the image.

In FIG. 6, the image segmentation function 172 (FIG. 5) begins at astart state 180 and proceeds to filter the image at a state 182. Theedge is the most difficult part of the object to select accurately. Edgepixels may be thought of as either the maxima of the first derivative orthe zeros of the second derivative (non-saddle inflection points) ofintensity with respect to X and Y. The purpose of the filter is toincrease the edge gradient. The larger edge gradients in the filteredimage result in feature extraction more insensitive to thresholdselection that follows filtering.

Presently, a convolution filter is used to filter the image. Theconvolution filter, or kernel, is loaded by the host processor 112(FIG. 1) into the real-time convolver (RTC) 138 (FIG. 4). Theconvolution is well-understood and may be concisely defined as follows:

    u.sub.i,j =c*g.sub.i,j                                     (2)

where:

u_(i),j is the pixel at the ith row and jth column of the filteredimage;

c is the convolution filter;

g_(i),j is the pixel at the ith row and jth column of the originaldigital image; and

* is the convolution operator.

In the presently preferred embodiment, the convolution filter is a 4×4kernel. The size of the kernel is limited by the presently preferredimage processor 110. However, it is expected that larger kernels mayprovide a better degree of filtering. The preferred convolution kernel,comprising a set of weights, that is used to enhance the contrast andincrease the edge gradient is as follows: ##EQU2## The convolution thatuses the above kernel is the equivalent of the following equation:##EQU3## where: u_(i),j is the pixel at the ith row and jth column ofthe filtered image; and

g_(i),j is the pixel at the ith row and jth column of the originaldigital image.

The result of the convolution is divided by 2 in the RTC 138 (bit shiftright one). If a pixel is >255 the ALU 136 sets it to 255 and if it is<0 it is set to 0. This kernel averages the images on a 2×2 scale (the3s in the kernel) and sharpens them on 4×4 scale (the negative numbersin the outer positions). The positive is in the outer positions alsoincrease the averaging effect. Convolution with this kernel and thesubsequent bit shift would cause an image of a single, nonzero intensityto double in brightness. This effect increases the contrast in imageswith multiple intensities.

The filtered image facilitates the use of a relatively simple thresholdselection method. The gradient, however, is not great enough to selectall nuclei with the same threshold. A second selection, using animproved threshold, is required. The sharpening effect of the filter andthe second threshold selection are next demonstrated on the samplenucleus shown in FIG. 3.

FIG. 7 is the three-dimensional plot of the convolved digital image of aDAPI stained nucleus and FIG. 8 shows the image separation applied tothe object 120. The original, shade-corrected image (e.g., FIG. 3) isused to compile final intensity parameters as the features from theconvolved image are sorted the second time (states 196, 200). Intensityparameters for calculating the second threshold are compiled from thefiltered image during the first sort 186. All nuclei are first selectedusing a single threshold (not shown). The estimated average intensity ofeach convolved nucleus is then calculated and used to find an improvedthreshold. In FIG. 7, the estimated average intensity of 190, indicatedat 214, and the threshold of 100, indicated at 216, are overlayed on theplot of the example convolved object 210. In FIG. 8, an overlay 218 onthe original object 120 shows the final selection of the nucleusresulting from this threshold; all pixels inside the overlay 218 arepart of the nucleus.

Referring to FIG. 6, after filtering the image at state 182, the systemmoves to a state 184 wherein the first threshold is obtained by findingthe first maximum in the histogram, obtained from the histogram/featureextractor 134 (FIG. 4), and adding a constant. Proceeding to state 186,the thresholds for the second selection are calculated by subtracting aconstant from the average intensity calculated for each object in thefirst selection. The lower limit for the second threshold is two-thirdsof the first threshold.

By way of an overview, object selection is accomplished by extractingthe feature pixels into an array and sorting the array onto a datastructure (Sort I). At state 188, given the intensities defining thefeature pixels, the histogram/feature extractor 134 (FIG. 4) of theimage processor 110 performs feature extraction by returning a streakarray, at state 192, which contains all positions of the correspondingpixels in a compressed form. Each time this occurs, i.e., another objectis found at state 190, the array must be sorted by the host processor112 to select individual objects.

Moving from state 192 to state 194, the second feature extraction isperformed in area-of-interest (AOI) mode on the image processor 110,which is significantly faster than normal mode. An AOI window is chosenby adding a border around the circumscribed rectangle, defined for eachobject in the first selection by its minimum and maximum x and y pixelpositions. Each nucleus is isolated from its neighbors by ignoringobjects touching the AOI border and comparing the remaining objects tothe original. To further speed execution, continuing to state 194, thesecond sort, Sort II, is carried out in parallel by initiating featureextraction of the next AOI before beginning the sort.

After each feature extraction, the pixel locations are sorted by thehost processor 112 onto a data structure (not shown) representing theobjects as shown in FIG. 6 at the states 196, 200, which both refer tothe Sort II function. (The state 198 performs the same function as state192, discussed above.) Sort II is further discussed with reference toFIG. 9. After all objects have been separated from the image, the imagesegmentation function terminates at an end state 246.

FIG. 9 illustrates the control flow for the Sort II function (a portionof the image segmentation function shown in FIG. 6) that sorts imagefeatures, or streaks, into objects beginning at a start state 222. Theflow diagram of FIG. 9 is the functional equivalent of the source codefor Sort II, which is included in the attached Microfiche Appendix. Itshould be observed that Sort II essentially incorporates the Sort Istate 186 (FIG. 6) used in the first selection step of the imagesegmentation function; however, Sort I does not include the hole fillingstep as defined in Sort II.

The Sort II function operates on an object data structure, stored in theRam memory of the host processor 112 (FIG. 1), that relates individualstreaks (from the image processor 110) into objects. The data structure,called the object list, is a circular, doubly-linked list of objectstructures, each pointing to a doubly-linked stack of the lines definingthe spatial extent of the object. Moving to a state 224, the first pixelposition data are provided in the form of a streak array by the HF 134(FIG. 4). A streak consists of the three values, x and y start locationand the length (l), denoting a horizontal set of contiguous featurepixels. This form of data compression has also been called "run lengthencoding" or "chord encoding".

Next, the object list is initialized at a state 226 by placing the firststreak into a first object structure. A line structure consists of thex, y, and l of one streak and pointers to the next and last lines on thestack. Each object structure contains descriptive information, pointersto the next and last objects in the list, and a pointer to the linestack. The descriptive information includes the minimum and maximum xand y values, object number, integrated intensity, area, and the sum ofthe squares and standard deviation of the pixel intensity. Thesevariables, with the exception of the standard deviation, are calculatedas the streaks are sorted. The sum of the squares is used to calculatethe standard deviation after the sort is completed.

Now, subsequent streaks are obtained from the HF 134 at a state 228. Atthe decision state 230, it is determined whether the new streak overlapsany object already contained in the object list. The Sort II functiongroups all connected streaks into the same object. Two streaks belong tothe same object if any pixel from the first is eight-connected to anypixel from the second. With this definition, two feature pixels areconnected if the first is one of the eight nearest neighbors of thesecond. Let one streak be x1, y1, l1 and the other be x2, y2, l2. Thesetwo streaks are connected if .linevert split.y1-y2.linevert split.=1,i.e., the streaks are in adjacent rows, and if the following conditionis satisfied:

    (x1<=x2+l2) and (x2<=x1+l1)                                (5)

Equation (5) mathematically describes the notion of columnar overlapbetween streaks. Thus, if at the decision state 230, it is determinedthat the new streak does not overlap any other streak in the objectlist, then the Sort II function creates a new object structure and linksit into the object list, as shown in FIG. 9 at state 232. The functionthen proceeds to a state 234 to determine whether another streak isavailable from the HF 134.

Now, returning in the discussion to state 230, if overlapping streaksare found, the function continues to a decision state 236, wherein atest is made as to whether the new streak overlaps two streaks in thesame object.

Most nuclei contain considerable detail and create images with internalintensity variations and edges. The sharpening filter accentuates thisdetail, sometimes causing areas within the nucleus to fall below thethreshold. This results in holes in the feature data representing thenucleus. These holes are filled during the second sort after moving fromthe state 236 to a state 238.

Filling the holes requires searching in both directions on the linestacks and is the reason for making them doubly linked. A hole isencountered when the streak being sorted is connected to more than oneline in the same object. The pair of line structures to which itconnects also define the entry point for the hole filling routine. Theholes are filled by finding all connected hole streaks and combining thepairs of line structures defining each. The set of contiguous pixelsbetween two line structures of the same y value define a hole streak ifthey are connected to the entry hole streak.

Two hole streaks are connected if any pixel from the first isfour-connected to any pixel from the second. Two pixels arefour-connected if the first is one of the four nearest neighbors of thesecond. This definition differs from eight-connected by excluding thefour closest diagonal pixels. The search for all connected hole streaksuses a stack, consisting of pointers to the first of the pair of linestructures defining each one. The entry hole streak is placed on thestack and the search is begun. A hole streak is removed from the stackand closed. Then all line structure pairs with y positions one less andone greater are checked and added to the stack if they define a holestreak. The cycle repeats until the stack is empty and the hole isfilled. Once the hole is filled, the function proceeds to decision state234.

Otherwise, if at the state 236 it is decided that the new streak doesnot overlap a second streak in the overlapping object, the Sort IIfunction moves to a decision state 240 to query whether the new streakoverlaps another object. If a streak connects with lines from twoobjects, they are joined at a state 242. The streak array is orderedleft to right and then top to bottom, with the x-y origin at the upperleft corner of the image. This order is utilized to speed sorting bymaintaining object list order in increasing x and dividing the list intoactive and inactive parts. An object becomes inactive when its greatesty is two or more less than the streak being sorted. Only the line stacksof active objects with appropriate x extrema are checked forconnectivity. After adding the streak and joining the objects in theobject list, Sort II moves to the decision state 234 as discussedpreviously.

From the decision state 240, if it is determined that the new streakdoes not overlap with two objects, the Sort II function proceeds to astate 244 to add the streak to the overlapping object, and then moves todecision state 234. The cycle of processing a new streak then continuesto the state 228, if one exists. Otherwise, the Sort II functionterminates at an end state 246.

Applications of the Image Cytometer

The device of the present invention makes fully automated,operator-independent image cytometry practical for use with attachedcells. The scanning cytometer recognizes and measures parameters oflabeled (e.g., fluorescent-stained) cell features of attached cells. Thedevice locates each cell in each image and computes quantitativeinformation about the amount of labeled cellular substance and its shapeand distribution. For example, the DNA content and distribution, andnuclear shape parameters, can now be identified and recorded with thedevice of the present invention. The device and methods describedherein, which are used, for example, to measure DNA content and cellnucleus parameters, are equally applicable to other cell features andcomponents.

The present invention is not limited to use in scanning DNA or nuclearmaterial. Other applications include, without limitation, ploidyanalysis of cells; analysis of cellular components including organellesand plastids; analysis of specific cell-derived molecules, such asmacromolecules; and improved analysis of cells or other materials,whether biological, organic or inorganic, by the presently-disclosedimage analysis techniques.

In addition, accurate recognition of the cell feature in the image is animportant step in achieving a fully automated operation. Others haveused simple intensity thresholding (which correctly recognizes, at best,about 85% of the cells analyzed) or mathematical morphology (recognitionrate unknown; see, e.g., C. J. Moran, A Morphological Transformation forSharpening Edges of Features before Segmentation, Computer Vision,Graphics, and Image Processing, Vol. 49, pp. 85-94, 1990) for imagesegmentation and recognition. The presently-disclosed method, whichyields a much higher accuracy recognition rate, is summarized asfollows:

    ACQUIRE IMAGE==>FILTER==>THRESHOLD

More detailed discussion of this method is provided herein. Also, theselection is refined by a second threshold for each individualobject--for example, each individual nucleus.

EXAMPLE I Cytometry of Attached Cells

The presently-disclosed device and method are preferably used to analyzefluorescent-stained cell features. Other investigators have had limitedsuccess with semi-automated DNA content from Feulgen-stained cells;further, none have presented DNA content data from more than 2,500stained cells, and typically, much less. There are many availablefluorescent dyes that stain other cellular features, but very fewdensitometric stains, such as Feulgen, can be used to quantitatecellular substances. Finally, the methods utilized for densitometricimage cytometry do not necessarily apply to fluorescence imagecytometry.

While the preferred embodiment utilizes the fluorescent stain DAPI, itshould be appreciated by those skilled in the art that other stains orlabeling means may be effectively utilized, such as antibodies taggedwith fluorescent or chemiluminescent moieties. Examples of fluorescentstains that may be used with live cells include DAPI and Hoechst (albeitthe latter is somewhat more toxic than DAPI). U.S. Pat. Nos. 4,906,561and 4,668,618 discuss the use of DAPI and are incorporated herein byreference. Thioflavin T and thiazole orange are fluorescent stainsdescribed in U.S. Pat. No. 4,957,870, which is incorporated herein byreference. Xanthene dyes are disclosed in U.S. Pat. No. 4,933,471, whilefluorescently-tagged antibodies are discussed in U.S. Pat. No.4,983,359; these patents are also incorporated herein by reference.Other fluorescent stains and methods of using same are described in U.S.Pat. Nos. 4,959,301 and 4,987,086, which are incorporated herein byreference. Examples of other fluorescent stains may be found in thecurrent catalog from Molecular Probes of Eugene, Oreg.

The disclosed invention could also be adapted for use with DNA-specific,densitometric, or other stains such as Feulgen Azure A, chromogen,methyl green, immunohistochemical stains, or ionic stains, for example,albeit they are much less preferred, as fluorescent labels provideadvantages over the use of non-fluorescent ones. (See, e.g., U.S. Pat.Nos. 4,998,284 and 5,016,283, which discuss alternative staining means.)

A. Cell Culture and Specimen Preparation

Cells were prepared for analysis as follows. Normal neonatal foreskinfibroblasts were plated on washed, autoclaved microscope slides. (Thecells were obtained form Dr. Robert Hoffman and Dr. Gilbert Jones, Univ.of Calif., San Diego.) The cells were maintained in Eagle's minimalessential medium with Earle's salts (Irvine Scientific, Irvine, Calif.),supplemented with 10% fetal bovine serum, 100 μg/ml gentamicin, and 0.26mg/ml L-glutamine (final concentrations), in a humidified 5% CO₂incubator at 37° C. When the cells reached about 50% confluence theslides were washed in physiologic buffered saline (PBS), fixed for onehour in 4% paraformaldehyde in PBS pH 7.2-7.4, and stained for one hourin DAPI solution. The staining solution consisted of 50 ng/ml4',6-diamidino-2-phenylindole dihydrochloride (DAPI), 10 mM TRIS, 10 mMEDTA, 100 mM NaCl, and 1% 2-mercaptoethanol. (See S. Hamada and S.Fujita, Histochem. 79: 219-226, 1983, which is incorporated herein byreference.) After staining, the slides were removed, covered with a fewdrops of the DAPI solution, and sealed with 22×50 mm coverslips and nailpolish. This creates a preparation with an excess of staining solutionsealed in with the cells, which contributes to stable fluorescence andrecovery after photobleaching. The P5 cell line used for demonstrationin FIG. 3 (and Figures based thereon, i.e., FIGS. 7 and 8) was preparedthe same way. (The P5 cell line is a culture of SV40-transformed skinfibroblasts that are murine in origin, obtained from Dr. Robert Hoffmanof the University of California, San Diego, Calif.) This cell linepreparation was useful for months, with its experimental longevity beinglargely dependent upon the quality of the seal between the coverslip andthe slide.

B. DNA Content Results

The results shown in FIG. 10 are an example of the data that can berecorded with this scanning cytometer. These data were obtained in sevenhours from 47,726 cells in a 22 mm×50 mm area by analyzing 8,062 images.Relative DNA content, which is proportional to the integrated intensity,is shown on the horizontal axis. The number of nuclei and the averageprojected area of those nuclei are shown on the vertical axes. Two setsof plots, corresponding to uncorrected (plain) 248, 252 and corrected(bold) 250, 254 data, are shown. The uncorrected and corrected DNAhistograms each exhibit two distinct peaks. The peaks on the left 248,250 correspond to the image objects containing a basal DNA content. Thesmaller peaks on the right 249, 251 correspond to the objects containingdouble the basal DNA content. These peaks are referred to as the 2n and4n peaks, where n is the number of chromosomes and a normal cellcontains a pair of each chromosome. Cells about to divide contain twicethe DNA content of resting cells. This relationship is exhibited in thecorrected histogram, in which the 2n peak is centered at a relative DNAcontent of approximately 33 and the 4n peak at approximately 66. Thecells between the two peaks are in S phase, synthesizing DNA. To createthe DNA content data, the integrated intensity of each nucleus wasdivided by a scaling constant and the result rounded. The number ofnuclei at each integer intensity was then summed to create thehistogram.

The DNA content histogram is, arguably, subjectively similar to the dataacquired with flow cytometry. The flow cytometer uses a single sensor,usually a photomultiplier tube, to measure the fluorescent intensity ofcells. The advantage of the photomultiplier tube is increased dynamicrange. The disadvantage is limited morphometry. While it can performsome morphometry based on scatter to forward- and side-mounted sensors,it does not have the hundreds of sensors used for each nucleus in thedata acquired here. This is an example of the kind of morphometric datathat are more difficult to obtain with a flow cytometer because of itslow spatial resolution.

Another difference between these two techniques is the fact that thescanning cytometry data contains mitotic cells in both the 2n and 4npeaks, whereas only the 4n peak contains mitotic figures in flowcytometry data. In flow cytometry, the 2n peak is usually referred to asthe G₁ or G₀ /G₁ peak, signifying cells in the resting phase of the celldivision cycle (G₁) or not in the division cycle at all (G₀). The 4npeak is called the G₂ +M peak in flow cytometry, signifying cells thathave completely duplicated their DNA or are in mitosis. The 4n or G₂ +Mpeak contains all the mitotic cells, because at the early stages of DNAseparation, the spatial discrimination of flow cytometry is not greatenough to distinguish the two sets of DNA in the same cell. In scanningcytometry, however, as soon as the DNA separates during anaphase ofmitosis, the computer "sees" two separate entities each with the 2n DNAcontent. Because the scanning cytometer in this example recognizes onlythe DNA, and not the cell, it does not distinguish whether two groups ofchromosomes are in the same cell or different cells.

Comparison of the coefficients of variation of the 2n peaks is commonlyused to evaluate relative system performance for DNA contentmeasurements. Use of the coefficient of variation is based on theassumption that all cells in 2n contain the same amount of DNA. Althoughthe width of the 2n peak can be affected by cell type, specimenpreparation and staining methods, it provides an estimate of thefluorometric precision. The coefficient of variation of the 2n peak inthe corrected data is 7.6%. The coefficients of variation reported bythe investigators who used simple intensity thresholding for recognitionranged from 11.5% to 12.7%. (See, e.g., T. Takamatsu, et al., ActaHistochem. Cytochem. 19: 61-71, 1986.) This is consistent with thecoefficients of variation generally associated with image cytometry.(See, e.g., G. L. Wied, et al., Human Pathology 20: 549-571, 1989.) Theanalogous coefficients of variation from flow cytometry range from 2% to8%. Clearly, then, the precision of the DNA content histogram presentedhere represents a significant improvement for fluorescence imagecytometry.

C. Discrimination of the Mitotic Figures by Area

Mitotic figures in DAPI stained cell monolayers are easily identifiedunder the microscope by looking for highly condensed, brightlyfluorescing nuclei. The fact that mitotic nuclei are so easilyidentified by the human observer raises the question of how easily theymight be automatically identified by computer. FIG. 11 shows a 3Dhistogram of data produced according to the present invention,specifically, the number of nuclei vs. their area and DNA content. The2n peaks 256 and the 4n peaks 258 are visibly separated from each otherfrom top to bottom in the figure. The S-phase nuclei are also visible inthe valley between (as indicated by reference letter "S" on theright-hand side of the histogram). On the left side, two distinctlyseparate peaks 257 can be seen. These peaks are likely to represent themitotic figures. Those objects in the 2n mitotic peak each representone-half of a mitotic figure. The fact that these are visible asseparate peaks raises the possibility of applying a curve fit to thedata and automatically locating the mitotic figures.

Note that both mitotic figure peaks 257 are spread out in the directionof DNA content. The mitotic FIGS. 257 should contain either the 2n or 4nDNA content. This spreading of the doubled DNA content mitotic figuresinto S and the single DNA content mitotic figures into the regionrepresenting less than the 2n DNA content is likely an artifact. Thisexample of photometric error, explained further below, is due to thefact that digitization to 8 bits is not adequate to measure the range ofintensities present in these preparations. The brightest nuclei arebrighter than the maximum intensity that can be measured by the cameraat its settings for this experiment. This explanation for the spreadingof the mitotic peaks is further substantiated by area vs. DNA contentrelationship exhibited within each. As the area decreases the measuredDNA content, or integrated intensity, also decreases. This is asexpected for a photometric clipping error. The more condensed thenucleus, the brighter its average intensity, and the greater the amountof light above the maximum that can be measured by the system. Withelimination of this error, the mitotic peaks become taller and moredistinct, improving the likelihood of automatic identification.

EXAMPLE II Cytometry of Live Cells, and Toxicity Studies

Development of scanning cytometry of growing cell monolayers requiresattention to cell culture chamber design, the effects of the measurementsystem itself on cell growth, and the reliability of identifying eachcell from scan to scan. The cell culture chamber must provide both anoptimal environment for growth and high grade optical accessibility forimaging. The fluorescent dye and excitation light used to create theimage must not appreciably affect cell growth. And finally, the intervalbetween scans must be short enough to unambiguously identify eachmigrating cell, and its daughters after division, for the duration ofthe experiment.

The cell culture chamber must allow maintenance of temperature, pH,osmolarity, nutrient concentration, and sterility. (See N M McKenna andY. -L. Wang, "Culturing Cells on the Microscope Stage," in FluorescenceMicroscopy of Living Cells in Culture, Part A, Methods in Cell Biology,vol. 29, Y. -L. Wang and D. Lansing Taylor eds., Academic Press, SanDiego, 1989 (see esp. p.108), which is incorporated herein byreference.) Temperature is controlled either by immersion of a sensor inthe medium or placement just outside the chamber, while pH is controlledby either gas or chemical buffering. Chambers with medium exposed to gasusually utilize CO₂ buffering while those without gas exposure use achemical buffer such as HEPES. Osmolarity control and nutrient supplyare maintained by medium changes. Sterility must be maintained duringthe transfer of cells to the chamber and while the chamber remains onthe stage. Sterility is easiest to achieve if a minimum of chambermanipulation is required after autoclaving and during introduction ofthe cells.

The presently preferred image cytometer 100 (FIG. 1) utilizes an uprightmicroscope and short working distance objectives for optimal opticalquality. Short working distance objectives may require the use of arelatively thin chamber. The problems of humidity control andcondensation on the microscope objectives can be avoided by use of aclosed, HEPES buffered, continuously perfused chamber. The chamberdesign thus consists of a glass slide and coverslip of equal rectangulardimensions held 250 μm apart by a retainer made of teflon. This retainermay contain access ports for the input and output of medium and theplacement of a thermistor type temperature probe. Upper and loweraluminum rectangular frames hold the glass pieces in the teflon retainerwith enough pressure to create a seal. A thin film of vacuum grease maybe applied between the teflon and glass pieces if necessary. All mediuminfusion will be through the teflon retainer and will contact only theglass once inside the chamber to avoid metallic ion toxicity.Temperature is controlled by use of a probe in direct contact with theculture medium and a heating element in the base plate of the stage. Thedesign allows for assembly prior to autoclaving to minimize the kind ofhandling that compromises sterility. Cells are introduced by infusionand infusion is stopped long enough for cell attachment. This designwill simplify handling and facilitate multi-day microscope stageculturing.

Scanning cytometry is dependent on the use of a fluorescent dye tocreate images simple enough for computer analysis. To determine thepotential for scanning cytometry of live cells, toxicity assays wereperformed with DAPI on normal human foreskin fibroblasts and atransformed 3T3 cell line. (Useful 3T3 cell lines are readily available;for example, one such cell line is the 3T3(A31) cell line, ATCC No. CRL6588, available from the American Type Cell Culture, Rockville, Md.)DAPI stains the DNA of live, as well as fixed, cells. Although thestaining is slightly less intense with live cells, it is sufficient foranalysis by scanning cytometry. The purpose of the assays was todetermine the concentrations at which DAPI begins to affect the growthrate of cells. The definition of toxicity used here includes metabolicchanges that might alter the grow rate of cells and is not limitedrequirement of cell death. The purpose of this definition is to includeany effect that the fluorescent dye could have on the parameters thatmight be measured by scanning cytometry. Ideally, the instrument shouldnot cause any changes in the object it is measuring.

A growth rate assay, utilizing the Coulter counter, was performed on theforeskin fibroblasts. This assay was carried out by plating cells on 7groups of P-150 culture plates (Nunclon #1-68381, 150×20, available fromMyriad Industries, San Diego, Calif.). These consisted of a controlgroup with no dye and 6 groups at concentrations of 10 ng/ml, 50 ng/ml,100 ng/ml, 500 ng/ml, 1,000 ng/ml, and 10,000 ng/ml of DAPI. Prior tostarting the cultures, these concentrations were added to aliquots of acommon batch of Eagle's minimal essential medium with Earle's salts,supplemented with 10% fetal bovine serum, 100 μg/ml gentamicin, and 0.26mg/ml L-glutamine (final concentrations). Each group consisted of fourreplicate plates for a total of 140 and the assay was begun by plating3000 cells on each.

The results of this assay are shown in FIG. 12. FIG. 12 shows that whileno toxicity occurs at 10 ng/ml, 50 ng/ml, and 100 ng/ml, metabolictoxicity does occur at 500 ng/ml and higher concentrations. The growthrate begins to be affected for 500 ng/ml between days 5 and 8. Thus, fora 5 day experiment, toxicity begins at a concentration of DAPI 10 timesgreater than that used for nuclear staining.

The second study, a clonal assay, was carried out on 3T3 cells. Thecells were grown under the same conditions as for the fibroblast assay.The cultures were begun at 500 cells/plate with 4 replicate plates foreach of the 6 DAPI concentrations and control. On day one, over 90% ofthe colonies were single cells. The cells were cultured for 10 days,rinsed with PBS, and fixed and stained with a mixture of 20% methanol,10% formalin, and 2% crystal violet in water. The visible clones on eachplate were counted and the mean and standard deviation of each set offour are shown in Table II below:

                  TABLE II                                                        ______________________________________                                        DAPI Clonal Toxicity Assay/3T3 Cell Line                                      ______________________________________                                        [DAPI] ng/ml:                                                                           0      10     50   100  500  1,000 10,000                           Number of 265    275    283  284  279  315   0                                Clones:                                                                       Standard  ±15 ±36 ±24                                                                             ±22                                                                             ±13                                                                             ±39                                                                              ±0                            Deviation:                                                                    ______________________________________                                    

These data show no significant difference in clone number until a DAPIconcentration of 10,000 ng/ml, at which no clones existed. Under phasecontrast microscopy, no attached cells were seen at this dyeconcentration. This implies that the toxicity for 3T3 cells occurs at ahigher concentration than for the normal fibroblasts. These studies,however, are difficult to compare because of the different techniquesutilized. Note in particular that at concentrations of 1 μg/ml and 500ng/ml in the fibroblast assay the decrease in growth rate was timedependent. This time dependent effect would not be measured in a clonalassay. By the time growth rate was slowed (day 5), colony size wouldhave been greater than the threshold required for counting and wouldhave no effect. These studies illustrate, however, that DAPI toxicity isnot likely to be a limiting problem for scanning cytometry of livecells.

Use of the presently disclosed methodology provides solutions for someof the problems inhibiting the use of images for automated cytometry.For the first time, nuclear recognition accurate and fast enough forfully automated operation has made measurement of 10⁴ -10⁵ attachedcells possible. Accurate recognition contributed to identification ofthe area-dependent photometric error caused by imperfect lightsensitivity. The solution to this error, combined with accuraterecognition, resulted in DNA content data approaching the precision offlow cytometry. This work represents the realization of one goal forimage cytometry: the measurement of nuclei in cell monolayers.

In spite of the improvement in photometric precision over other imagecytometry reports, the achievement 2n (or G₁ +M) coefficients ofvariation comparable to flow cytometry will probably require improvementin several scanning cytometry components. The magnitude of some of theerrors caused by arc lamp instability, narrow depth of field, limitedintrascene dynamic range, and photobleaching was presented, along withmethods for improvement, in the incorporated dissertation. With theimprovements presented herein, scanning cytometry may become as precisea fluorometric tool as flow cytometry.

The use of scanning cytometry on live cells is compelling because cellsdo not have to be suspended and repeated scanning should allow temporalmeasurements of a group of cells on a cell-by-cell basis. The scanningcytometer presented here is capable of a scanning interval of 30 minuteson 10³ -10⁴ cells. This interval may allow resolution of the major cellcycle phases: G₀, G₁, S, G₂ and M.

The image cytometer itself could affect cell growth through dye toxicityor phototoxicity. The DAPI toxicity assays presented herein show thatcell growth is not affected until a dye concentration of 100-500 ng/ml,about 10 times greater than the 10-50 ng/ml required for staining.Shuttered image acquisition, combined with phase contrast autofocus,minimizes the amount of excitation light exposure and limitphototoxicity. The comparison of fifteen autofocus functions referred toherein (and detailed in the incorporated dissertation), yielded two thatshould be useful for autofocus with phase contrast microscopy. The doseresponse of cell growth to the excitation light, however, is still underinvestigation. If phototoxicity is not significant, image cytometry islikely to become a powerful tool for studies involving live cells.

It is also possible to decrease or minimize light intensity via use of amore sensitive camera or other image sensor. Preferably, such a devicehas high sensitivity, high resolution, and limited geometric distortion.In addition, fiber optic coupling and video output are preferredattributes. The ICCD-1381F Intensified CCD Camera (Video ScopeInternational, Ltd., Washington, D.C.) is one such example of a moresensitive image sensor, which may be applied to either low light levelfluorescence or standard low light level analysis/surveillance.

EXAMPLE III Correction of Photometric Offset Error

FIG. 10 also demonstrates the correction for an error caused by thecombination of two image cytometry characteristics: imperfectsensitivity and intensity measured by multiple sensors. This error canbe demonstrated by assuming the existence of two nuclei of equal DNAcontents and different projected areas, measured by different numbers ofsensors. Suppose nuclei n1 and n2 have areas of a1 and a2 pixelsrespectively. Let it1 and it2 be the average true fluorescence of thesmaller subsampled regions of n1 and n2. If is is the intensity lost dueto imperfect sensitivity then ip1=it1--is and ip2=it2-is are the averagepixel intensities. Because both nuclei have the same DNA content, theirtrue integrated intensities are also equal and a1 (it1)=a2 (it2). Theintegrated intensities of n1 and n2 calculated from the image are asfollows:

    I1=a1(it1-is) and I2=a2(it2-is)                            (6)

Substituting it2=a1it1/a2 yields I2=a1it1-a2is. Combining equationsgives the equation:

    I1=I2+(a2-a1)is                                            (7)

The relative intensities therefore differ by an amount that depends onthe difference in area and the amount of unmeasured light per pixel.Nuclei with larger area and the same DNA content have lower integratedintensities. The larger the area, the more the DNA content isunderestimated. This dependence on area is exhibited by the negativeslope of the area curve in the region of the G₁ peak in the uncorrecteddata. Another effect of the unmeasured intensity is a shift in theuncorrected histogram to the left, resulting in a G₂ peak centered atgreater than twice the intensity of the G₁ peak.

The photometric offset is due largely to imperfect camera sensitivity.This offset also varies with the gain and level settings on the cameraand image processor. An indirect method for determining the offset errorwas utilized for the data shown in FIG. 10. This method is based on theassumption that area is independent of DNA content in the region of the2n peak. These image objects all have the same DNA content and errors inmeasuring that content should be random (the mitotic figures areexceptions, see discussion supra). A random relationship to area wouldyield a zero slope in the area curve in the 2n region. The integratedintensities were corrected by adding back an offset value for eachpixel. This constant was assumed to be correct when the slope of areacurve in the G₁ region became zero. Note that the G₁ peak is narrowerand taller, suggesting a lower coefficient of variation, and that theleft shift of the peaks is corrected.

A second, direct method for measuring the photometric offset should alsobe possible. Before an experiment, after the proper light intensity,gain, and offset have been determined, a calibration curve usingprecalibrated neutral density filters could be plotted. The averagedigital intensity of the shade-corrected flat field image would beplotted as a function of filter transmittance. The y-intercept of thislinear function should provide a reasonably accurate system offsetmeasurement that could then be added to each pixel during analysis. Thefirst, indirect method allows data correction even after the imageprocessor and camera settings have been changed while the second, directmethod provides a completely data-independent method of correction.

Several other non-trivial improvements in operator-independent imagecytometry should be acknowledged, as they are important components ofthe present invention. For example, the increase in speed will yieldresults on fixed cells in more reasonable times and improve temporalresolution for live cell studies. The theoretical limit with the imageprocessor used here is about 0.25 s/image, or about an order ofmagnitude faster than current performance. Another suggestion providedby the present disclosure is the use of additional fluorescent dyes forrecognition of other cellular features. The methods described herein forenhancing contrast to improve recognition accuracy are also applicableto other cellular features.

Finally, general application of scanning cytometry to cell smears,histologic sections and cells that grow without contact inhibition, orin matrices, has been advanced by the present invention. While we arecontinuing to address various aspects of the wide scale implementationof scanning cytometry as a general tool for cell physiology, importantcomponents of scanning cytometry have been advanced by the workdisclosed herein.

The Arc Lamp Stabilization and Intensity Control System

Arc lamps exhibit short-term flicker, intermediate-term variation, andlong-term intensity decay. The short term flicker includes spatial andtemporal variations in intensity. The spatial variation is inherent inan arc lamp which produces an effect called arc wander, i.e., the arcmoves over time. Such arc wander results in undesirable changes inintensity thus reducing the quality of the magnified image.

Thus, having an arc lamp with smooth spatial and temporal intensitywould be an important step in microscopy. The present invention includesan arc lamp stabilization and intensity control system that provides thedesired results. Smoothing spatial variations in intensity is achievedvia use of a light scrambler, preferably embodied as an optical fiber.In addition, greater light can be applied to a specimen by employing theellipsoidal reflector of the present invention.

By using the light scrambler, a photodiode can be used to measureaverage intensity. The photodiode produces an input to a feedback systemthat varies the current to the arc lamp, i.e., intensity control. Thecombination of these various elements in a system results in a degree oflight source stability previously unattainable with arc lamps.

A schematic of one prior art microscope lighting system is illustratedin FIG. 13. In the following discussion many parts will include acommercial source in parentheses. However, it should be understood thatthese are currently preferred sources and that other sources and partsmay be substituted therefor.

In FIG. 13, an arc lamp 260 (OSRAM HBO 100W/2, Bulbtronics, Anaheim,Calif.) is provided as the light source for the optical components of amicroscope (e.g., microscope 102 shown in FIG. 1). The light generatedfrom the arc lamp 260 is indicated at 261. A reflector 262 (Nikon Inc.,Garden City, N.Y.) reflects light from one side of the arc lamp 260 backin the direction of the microscope. The direct light from the arc lamp260, and the light reflected from the reflector 262, is collected by aset of light source collector lenses 264 (Nikon Quartz Collector).

From the light source collector 264, the light travels through a lens266 (Nikon) (note that the reflector 262 and lens 266 are parts of thestandard Nikon Optiphot with Epifluorescence) and an exciter filter 268(365 DF 12, Omega Optical, Brattleboro, Vt.) to a dichroic mirror 270(DC 405 LP, Omega Optical). The light is reflected from the mirror 270so as to travel through an objective 274 (20x phase/fluorite, 85002,Nikon) onto a specimen 114 on the stage 103 (FIG. 1) of the microscope102.

The magnified image of the specimen 114 then travels back through theobjective 274, the dichroic mirror 270 and an emitter filter 272 (485 DF30, Omega Optical). The image is focused through two video cameraprojection lenses 276, 278 (TV 1×16, Nikon) and received by the videocamera 108.

The resulting image produced in the prior art microscope from the arclamp light source includes spatial and temporal variations in imageintensity that limit the capabilites of image separation as previouslydescribed. However, it has been shown that spatial variance in intensitycan be reduced by the use of a light scrambler.

Turning now to FIG. 14, a light scrambler is shown in a schematic of asecond prior art microscope lighting system. The light scramblercomprises three parts that are added to the system of FIG. 13 betweenthe arc lamp 260 and the light source collector lenses 264. First, FIG.14 differs from FIG. 13 by the insertion of a set of second light sourcecollector lenses 280 after the arc lamp 260. The collector lenses 280are typically identical to the light source collector lenses 264.Second, a set of fiber optic focusing lenses 282 is added after thecollector lenses 280. The fiber optic focusing lenses 282 are typicallyidentical to the light source collector lenses 264, but reversed.Finally, a length of optical fiber 284, for instance, 1 millimeterdiameter×1 meter, (77513, with connectors, 77573, Oriel Corp.,Stratford, Conn.) is inserted after the focusing lenses 282. The fiberis coiled one or more turns (three turns with the preferred opticalfiber) so that the light is randomly reflected, thereby producing asmooth spatial intensity at the light source collector lenses 264. Thelight from the focusing lenses 282 travels through the remainder of theoptical system as discussed with respect to FIG. 13.

The lenses 280, 282 function to collect and focus light into one end ofthe optical fiber 284. In traveling through the coiled fiber 284 thelight 261 is reflected many times off the internal wall of the fiber284. These multiple reflections scramble the image of the arc lightsource.

It should be observed that the lenses 262, 280 and 282 form onepreferred light scrambling configuration suggested by Technical VideoLimited of Woods Hole, Mass., but other configurations may be possible.

The presently preferred embodiment of the arc lamp stabilization andintensity control system of the present invention is illustrated in FIG.15. In this system, the arc lamp 260 generates light 261 into anellipsoidal reflector 286 (Photomax F/2 Reflector with AlMgF₂ Coating60113, Oriel Corp.). The light 261 is then reflected by a dichroicreflector 288 (350-450 nm, 60142, Oriel Corp.) into one end of theoptical fiber 284.

Light passes through the optical path previously described with respectto FIG. 14, i.e., onto the specimen 114 and through the objective 274 sothat the video camera 108 receives a magnified image of the specimen114.

Stabilization and intensity control is established through feedbackwhich will now be described. A photodiode 290 (SD-076-12-12-011, SiliconDetector Corp., Camarillo, Calif.) is placed between the dichroic mirror270 and the objective 274 so as to receive a portion of the light 261.The photodiode 290 is positioned off the optical axis far enough toavoid appearing in the image of the specimen 114. The photodiode 290produces an electrical signal that is amplified by a photodiodeamplifier 292 (which is further described with respect to FIG. 16). Theamplified signal is fed to a servo amplifier 294 (preferably the pulsewidth modulated (PWM) servo amplifier, model 220, available from CopleyControls Corp. of Newton, Mass.) that is controlled by the hostprocessor 112 via an analog-to-digital (A/D) input/output board such as,for example, model DT2801-A from Data Translations, Marlboro, Mass.

In the preferred embodiment, the photodiode amplifier 292 is adjusted todeliver a range of about 0 to 5 volts dependent on the normal intensityrange viewed by the photodiode 290. The servo amplifier 294 receives theoutput of the photodiode amplifier 292 and a reference voltage from thehost processor 112. The servo amplifier 294 then alters the current tothe arc lamp 260 in such a way as to maintain no, or very little,difference between the output voltage of the photodiode amplifier 292and the reference voltage. In the presently preferred embodiment, thecurrent across the line 300 is in the range of about 1.5 to 5.5 amperesfor a 100 watt arc lamp. This establishes a desired light intensity forthe specimen 114.

If intensity is changed from the host processor 112 by specifying a newreference voltage, the servo amplifier 294 will change its outputcurrent to alter intensity until the photodiode amplifier 292 outputvoltage matches the new reference voltage. As the arc lamp 260 changestemperature secondary to the new current, the servo amplifier 294 willmaintain constant intensity by changing its output current as required.Also, instability in the intensity of the arc lamp 260 attributed to thearc wander or arc lamp decay will be compensated by the system of thepresent invention.

FIG. 16 illustrates a schematic diagram of the preferred photodiodeamplifier 292 (power sources not shown) of the arc lamp stabilizationand intensity control system shown in FIG. 15. In FIG. 16, thephotodiode 290 produces a current from the light emitted by the arc lamp260 (FIG. 15). The signal thus generated is amplified by a differentialamplifier 304, such as, for example, an LF356 operational amplifier.

The gain of the amplifier 304 is controlled by a potentiometer 306. Thepotentiometer 306 is adjusted so that the voltage output by thephotodiode amplifier 292 falls within the range of servo amplifier inputvalues (FIG. 15). The offset of the amplifier 306 is also controlled bya potentiometer 308. The potentiometer 308 is adjusted so that athreshold photodiode voltage results in about zero voltage output fromthe amplifier 304.

The output signal of the amplifier 304 is fed to a buffer amplifier 310and then to the servo amplifier 294 (FIG. 15). The amplifier 310 buffersthe differential amplifier 304 from changes in impedance attributed tothe servo amplifier 294.

SUMMARY

Although the present invention includes an image cytometer for cellmeasurement, one skilled in the technology will recognize that there areother applications of the image segmentation described herein, Forinstance, the present invention could have an application to particlerecognition as known in the material sciences.

While the above detailed description has shown, described and pointedout the fundamental novel features of the invention as applied tovarious embodiments, it will be understood that various omissions andsubstitutions and changes in the form and details of the deviceillustrated may be made by those skilled in the art, without departingfrom the spirit and scope of the claimed invention.

What is claimed is:
 1. An automated method of separating an object froma background in an image comprising pixels, the method comprising thesteps of:selecting a set of weights in a digital filter, wherein atleast a portion of the weights is selected to enhance at least a portionof the object, said object portion characterized by a predeterminedmultispectral pattern comprising a plurality of spectral bands;transforming the image with the digital filter to enhance pixelsassociated with the predetermined multispectral pattern of the objectnot contained in the background so as to produce a transformed image,wherein the multispectral pattern comprises a plurality of edgegradients and non-edge gradients, wherein at least a portion of non-edgegradient values falls within the range of the edge gradient values, andwherein the weights are selected so as to enhance both the edge andnon-edge gradients which are included in the multispectral pattern,thereby allowing object identification with the use of said digitalfilter; and thresholding the transformed image, wherein the thresholdingstep includes extracting object features from the transformed image andsorting the extracted object features so as to separate the object fromthe background.
 2. The method of object separation defined in claim 1,wherein the digital filter comprises a convolution filter.
 3. The methodof object separation defined in claim 1, wherein the digital filtercomprises a Fourier filter.
 4. The method of image separation defined inclaim 1, wherein the thresholding step includes an object specificthreshold.
 5. The method of image separation defined in claim 1,additionally comprising the step of hole filling the object.
 6. Themethod of image separation defined in claim 1, wherein the object is acell.
 7. The method of image separation defined in claim 6, additionallycomprising the step of measuring cell nucleus parameters.
 8. The methodof image separation defined in claim 7, wherein the step of measuringcell nucleus parameters includes measuring the DNA content of the cell.9. The method of image separation defined in claim 1, wherein thedigital filter comprises an averaging portion of a first scale and asharpening and averaging portion of a second scale.
 10. The method ofimage separation defined in claim 9, wherein the averaging portion isperformed on a 2×2 scale, and the sharpening and averaging portion isperformed on a 4×4 scale.
 11. The method of image separation defined inclaim 1, wherein the background comprises a plurality of backgroundgradients and wherein at least a portion of the background gradientvalues falls within the range of the edge gradient values.
 12. Anoperator-independent image cytometer, comprising:a microscope, having amicroscope stage, for optically magnifying at least a portion of aspecimen having a plurality of cells; a camera for creating an image ofthe magnified specimen, wherein the image includes a background; meansfor digitizing the image so as to create a digital image; means fordigital filtering the digital image into a filtered image with a filtercomprising a set of weights, wherein at least a portion of the weightsis selected to enhance at least a portion of the cells, said portioncharacterized by a multispectral pattern comprising a plurality ofspectral bands so as to create contrast between the cells and thebackground, wherein the multispectral pattern comprises a plurality ofedge gradients and non-edge gradients, wherein at least a portion of thenon-edge gradient values falls within the range of edge gradient values,and wherein the weights are selected so as to enhance both the edge andnon-edge gradients which are included in the multispectral pattern; and,means for thresholding the filtered image.
 13. The image cytometerdefined in claim 12, wherein the digital filtering means includes aconvolution filter.
 14. The image cytometer defined in claim 12, whereinthe digital filtering means includes a Fourier filter.
 15. The imagecytometer defined in claim 12, additionally comprising means forcontrolling the microscope stage in an X,Y plane so that the specimencan be scanned in a sequence of fields.
 16. The image cytometer definedin claim 12, additionally comprising means for controlling themicroscope stage in a Z dimension so as to autofocus the microscope. 17.The image cytometer defined in claim 12, wherein the digitizing meansincludes a shade corrector.
 18. The image cytometer defined in claim 12,wherein the thresholding means determines a threshold intensity tosegment the cells from the background.
 19. The image cytometer definedin claim 12, wherein the filtered image comprises pixels indicative ofthe cells and the background, and wherein the thresholding meansadditionally includes means for averaging intensities of pixelsindicative of thresholded cells so as to eliminate a portion of thepixels from each thresholded cell.
 20. The image cytometer defined inclaim 12, additionally comprising means for hole filling the cells. 21.The image cytometer defined in claim 12, wherein the specimen is stainedwith a fluorescent dye.
 22. The image cytometer defined in claim 12,wherein the camera is a CCD camera.
 23. The image cytometer defined inclaim 12, wherein the cells are live cells.
 24. An operator-independentimage cytometer, comprising:a microscope for creating a magnified imageof a specimen; a stage controller for moving the specimen relative tothe microscope; a camera for acquiring the magnified image; and meansfor analyzing the acquired magnified image including a pattern filtercomprising a set of weights, wherein at least a portion of the weightsis selected to enhance at least a portion of the image, said portioncharacterized by a multispectral pattern comprising a plurality ofspectral bands, wherein the multispectral pattern comprises a pluralityof edge gradients and non-edge gradients, wherein at least a portion ofthe non-edge gradient values falls within the range of edge gradientvalues, and wherein the weights are selected so as to enhance both theedge and non-edge gradients which are included in the multispectralpattern.
 25. The operator-independent image cytometer defined in claim24, wherein the analyzing means further comprises:means for transformingthe image into a transformed image including object features having afirst set of intensities and background features having a second set ofintensities; and means for thresholding the transformed image so as toseparate the object features from the background features.
 26. An imageseparation system, comprising:means for acquiring a digital imagewherein the digital image comprises a plurality of pixels indicative ofa set of objects and a background; a digital filter comprising a set ofweights, wherein at least a portion of the weights is selected toenhance at least a portion of at least one object, said object portioncharacterized by a multispectral pattern comprising a plurality ofspectral bands, said filter receiving the digital image and producing afiltered image and wherein the multispectral pattern comprises aplurality of edge gradients and non-edge gradients, wherein at least aportion of the non-edge gradient values falls within the range of edgegradient values, and wherein the weights are selected so as to enhanceboth the edge and non-edge gradients which are included in themultispectral pattern; wherein each pixel of the digital image and thefiltered image has at least one associated numeric value correspondingto a characteristic parameter of the objects and the background; whereina difference between the numeric values associated with the pixels ofthe objects and the numeric values associated with the pixels of thebackground is greater in the filtered image than in the digital image; ahistogrammer producing a histogram of the filtered image, wherein thehistogram represents the filtered image numeric values; first thresholdmeans for determining a threshold numeric value from a histogram andseparating a set of thresholded objects from the filtered image; andsecond threshold means for determining a local threshold value for eachof the thresholded objects and selectively removing a pixel from eachthresholded object if the pixel has a filtered image numeric value thatis on a preselected side of the local threshold value in the preselectedrange of values.
 27. The image separation system defined in claim 26,wherein the characteristic parameter is indicative of intensity.
 28. Anautomated method of separating an object from a background in an imagecomprising pixels, the method comprising the steps of:selecting a set ofweights in a digital filter, wherein at least a portion of the weightsis selected to enhance at least a portion of the object, said objectportion characterized by a predetermined multispectral patterncomprising a plurality of spectral bands; transforming the image withthe digital filter to enhance pixels associated with the predeterminedmultispectral pattern of the object not contained in the background soas to produce a transformed image, wherein the multispectral patterncomprises a plurality of edge gradients and non-edge gradients, whereinthe background comprises a plurality of background gradients, wherein atleast a portion of the background gradient values falls within the rangeof the edge gradient values, and wherein the weights are selected so asto enhance both the edge and non-edge gradients which are included inthe multispectral pattern, thereby allowing object identification withthe use of said digital filter; and thresholding the transformed image,wherein the thresholding step includes extracting object features fromthe transformed image and sorting the extracted object features so as toseparate the object from the background.
 29. The method of imageseparation defined in claim 28, wherein at least a portion of thenon-edge gradient values falls within the range of the edge gradientvalues.